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Between 2012 and 2018, the U.S. Department of Education invested nearly $350 million in 22 Comprehensive Technical 
Assistance (TA) Centers operating across the nation. These Centers were charged with delivering TA that builds the capacity of 
state education agencies (SEAs) to support local educational agencies (LEAs) in improving student outcomes. Centers were given 
broad discretion in interpreting and enacting this mandate. This evaluation sought to address the open questions about how the 
Centers designed and implemented the TA, what challenges they encountered, and what outcomes they achieved. With 
thorough documentation of how this process played out, stakeholders will be in a better position to inform future program 
improvement. Key takeaways from the study include: 


e Overall, Centers and their TA recipients reported that the Centers’ TA improved the capacity of SEAs to meet their goals. 


e Centers shared similar approaches to the design and implementation of their TA. Those Center practices perceived to be 
instrumental to building capacity included: engaging a broad array of stakeholders to provide input on policy; providing 
products and tools for SEA staff to use as they took greater ownership of policy design and implementation; imparting 
organizational practices and structures resilient to SEA turnover and policy shifts; and flexibly adapting TA in response to 
changing priorities and needs. 


e Centers and their TA recipients pointed to a few areas for program improvement, including clarification of the Centers’ role 
and expected outcomes related to their work with LEAs, and further guidance for SEAs about how best to use the Centers. 


Since the 1990s, state education agencies (SEAs) have faced pressure to take on expanded roles and responsibilities in carrying 
out educational reforms. SEAs, however, often have limited resources to meet the new challenges they previously faced under 
No Child Left Behind and now face under the Every Student Succeeds Act (ESSA). Title Il of the Educational Technical Assistance 
Act of 2002 (Section 203)* authorized the Comprehensive Centers program. In 2012, the U.S. Department of Education 
awarded five-year grants, later extended to seven years, to 15 Regional Centers and 7 Content Centers under the program. 
These grants were awarded with the goal to “provide technical assistance to SEAs that builds their capacity to support local 
educational agencies (LEAs or districts) and schools, especially low-performing districts and schools; improve educational 
outcomes for all students; close achievement gaps; and improve the quality of instruction.” 


Regional Centers were required to build SEA capacity across seven federal priority areas, focusing on those priorities that 
addressed each state’s needs. The seven federal priority areas included: implementing college- and career-ready standards, 
developing highly effective teachers and leaders, turning around low-performing schools, ensuring school readiness and 
transitions to kindergarten of preschool children, building rigorous pathways from secondary school to college, identifying and 
scaling up innovative approaches to learning, and using data-based decision-making to improve instruction and outcomes.? In 
2016, assistance with ESSA implementation was added as a priority area for the Centers’ work. Regional Centers each served 
between one and seven U.S. states (or territories and freely associated states in the Pacific Region). For example, the Texas and 
California Comprehensive Centers each served one state, while the Northeast Comprehensive Center served Connecticut, 
Maine, Massachusetts, New Hampshire, New York, Rhode Island, and Vermont. Each of the seven Content Centers was required 
to deliver technical assistance (TA) nationally in one of the federal priority areas, using content expertise to build capacity of 
both the SEAs and the Regional Centers. (See Appendix A for the full list of Centers, along with the states and entities served by 
each Regional Center.) 


The Comprehensive Centers Program gave Centers considerable discretion in designing and implementing TA projects within 
the designated federal priority areas. While Centers were expected to build SEA capacity to lead or support LEAs in improving 
educational outcomes, the program recognized the wide variation in the priorities and existing capacity of SEAs, and urged 

Centers to develop TA plans based on “in-depth knowledge and understanding” of state needs and contexts. Content Centers 
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were required to draw on “research-based practices and emerging promising practices” in their specialized content areas,* but 
in general, Centers were not required to adopt specific designs of TA or capacity-building services.° (For two detailed examples 
of Centers’ TA projects, see Appendix E.) 


Given the many possible definitions of capacity building that could drive this work, this national evaluation sought to answer 
questions as to how Centers actually interpreted and enacted their broad mandate and whether they did so in similar ways. The 
evaluation approached these questions by examining Centers’ design and implementation processes, as well as their outcomes 
and challenges. Such a nuanced look will help the program better understand whether guidance to the Centers was sufficient to 
produce intended outcomes. The results of this evaluation will also help the program to identify areas where outcomes are not 
as intended and how program design might be modified to help address documented challenges. 


More specifically, there are three main “black boxes” that this evaluation sought to unpack (see Exhibit 1): (1) Design (how did 
Centers interpret and plan their work); (2) Implementation (how did Centers actually carry out their work); and (3) Outcomes 
(what results were achieved). Within each box, the evaluation addressed a narrower set of questions that are expected to be of 
interest given the broad program requirements, such as how Centers define capacity building and set goals, assess SEA needs, 
deliver TA, address challenges, and produce outcomes. To cover the full breadth of how Centers designed their work, the design 
component of the evaluation factored in the Centers’ work in all seven federal priority areas. The implementation and 
outcomes components, in contrast, focused only on Centers’ work in the early learning and teacher/leader effectiveness 
priority areas, to provide a more in-depth look while limiting burden on respondents. The implementation and outcomes 
components are thus skewed towards those Centers that had more work in those areas, although all Centers had projects in at 
least one of the two priority areas each year. 


Exhibit 1. Evaluation Questions 


Design 


¢ How did the Centers define capacity building in their theories of action? 
* How did the Centers assess the needs of their constituents and develop work 
plans to address those needs? 


Implementation 


* What strategies did the Centers employ to achieve their outcomes? 
* To what extent and how did the Centers collaborate with each other? 
* What challenges did Centers face and how did they respond? 


Outcomes 


* Did Centers achieve their expected capacity-building outcomes, and how did 
they know? 
* What strategies were perceived to be most effective and why? 


The bulk of this report will focus on presenting findings for the questions within each of the boxes in Exhibit 1. At the end of the 
report, we return to Exhibit 1 and fill it in with the specific findings, along with a set of lessons learned and recommendations 
that are synthesized from the findings. Exhibit 2 provides an overview of the data used to address the evaluation questions. It is 
important to emphasize that this evaluation is not able to measure the causal impacts of the Centers’ work and that much of 
the findings are based on the perceptions of Center staff and TA recipients identified by the Centers. The findings should 
therefore not be used to make summative judgments about the Comprehensive Centers program but rather as insights into 
how the program is playing out in practice and possible avenues to help it better meet program objectives in the future. 
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Exhibit 2. Data Sources 


The study results are based on a variety of data sources summarized below. Additional details about the data and study design 
can be found in Appendices B-C, with more information about the selection of the two priority areas on page B-2. Two detailed 
project profiles are provided in Appendix E. 


Surveys. A survey for Center staff and another for TA recipients were conducted in spring of 2015, 2016, and 2017. All Center staff 
who worked on projects in the two priority areas and key TA recipients identified by Centers (using guidelines provided by the 
evaluation team) were sampled. Center staff answered the same set of survey questions separately for each project in the two 
priority areas on which they had worked, while TA recipients answered a single, longer set of questions that combined ail projects 
relevant to them in the two priority areas. As a result, the Center staff survey was analyzed at the project level, and the TA 
recipient survey was analyzed at the respondent level. Final Center staff samples were 166 to 194 per year (response rates: 79% 
to 86%) and covered 123 to 149 projects per year. TA recipient samples were 232 to 246 (response rates: 67% to 68%) and 
covered 119 to 143 projects per year. 


Interviews. Annual interviews with Center staff and TA recipients featured protocols for design topics, implementation, and 
outcomes. Centers’ TA design across all priority areas were addressed in one group interview with each Center in 2015; 
participants included Center directors and lead TA providers. Implementation (2015 and 2016) and outcomes (2017) were 
addressed in project-focused interviews with Center staff and TA recipients. Centers selected 1-2 key projects in each of the two 
priority areas that provided in-depth illustrations of capacity building; identified staff working on these projects to participate in 
group interviews; and identified 1-2 TA recipients per project to be contacted for separate, individual telephone interviews. 
Overall, 55 projects were discussed in Center staff implementation interviews, of which 44 were also discussed in TA recipient 
interviews; 36 projects were discussed in Center staff outcomes interviews, of which 31 were also discussed in TA recipient 
interviews. 


Activity Reports. These reports provided a full account of projects that Centers conducted in the two priority areas. Collected 
every six months from October 2014 to 2017, the data included types of TA activities or products, service dates, and constituents. 
A total of 333 projects were represented over three reporting years. 


Documents. The evaluation team reviewed overview documents from each Center, including theories of action or logic models. 
Annual management plans and project documents relevant to interviews were collected each year. We also collected year 4 
Center evaluation reports prepared by the Center staff or their evaluators. 


1. HOW CENTERS DESIGNED THEIR TECHNICAL ASSISTANCE 


Capacity building is a broad concept, and federal policy provided some flexibility to Centers on how they interpreted it. It is 
therefore valuable to understand how Centers defined capacity building (and whether they did so in similar ways), particularly 
since their approach could shape subsequent behaviors related to TA planning, strategies, activities, and expected outcomes. 
To do so, the evaluation team primarily relied on interviews conducted in 2015. (More details on the literature about capacity 
building and specific findings can be found in Appendices A and D.) 


Centers had a similar view of capacity building, broadly defining it as strengthening the long-term ability of SEA staff and 
organizations to work with LEAs and improve educational outcomes. In interviews conducted in 2015 about the design of Centers’ 
TA, all 22 Centers identified SEA staff knowledge and skills as targets for capacity building, and all but one Center also identified 
organizational capacity building, policy development capacity, and/or policy implementation capacity. (See Appendix Exhibit D-1 
for detailed interview results.) 


e Knowledge and skills or human capacity building. Centers defined this type of capacity building as contributions to the TA 
recipients’ acquisition of new knowledge and more advanced skills, including management and leadership skills, content 
knowledge, or technical skills in areas such as policy design or data analysis. For example, one Center reported providing 
research and policy scans that enhanced SEA staff knowledge of principal training and pipeline development. 
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e = Organizational capacity building. Centers defined this type of capacity building as the creation of SEA organizational 
improvements through restructuring; through improved communication and coordination across staff and divisions or with 
other agencies; or through improvements in other processes, procedures, or use of resources. For example, one Center 
described assisting an SEA in developing the practice of conducting cross-divisional meetings whenever it developed 
strategic plans, bringing together staff who had not worked together previously. 


e = Building capacity for policy development or design. Centers defined this type of capacity as the enhancement of SEAs’ ability 
to develop state policy, including recommendations and advice to boards of education and the legislature. For example, 
one Center described an SEA’s development of new teacher standards using tools provided by the Center, including a 
framework and step-by-step guide to standards development. 


e = Building capacity for policy implementation. Centers also described capacity building that focused on enhancing SEAs’ ability 
to roll out or disseminate a policy reform or practice throughout the state, with improved outreach to LEAs and better 
ability to provide support, guidance, or training to LEAs implementing new policies. For example, one Center reported 
preparing SEA staff to lead training for LEA staff on the administration of new teacher evaluations to be used for 
instructional improvement. 


To strengthen these areas of SEA capacity, all but one Center reported in 2015 interviews that they sought to incorporate principles 
of capacity building into their TA, including fostering ownership, long-term change, and/or organizational process changes. (See 
Appendix Exhibit D-2 for detailed interview results.) 


e = Fostering ownership. Centers aimed to “gradually release,” or reduce their roles over time, encouraging SEAs to fully take 
over new practices. 


e = Fostering long-term change. Centers aimed to promote changes that were likely to be sustained by the SEA after the 
Centers’ work with the SEA ended and that could survive staff turnover and changing priorities. 


e = Fostering organizational process changes. Centers aimed to produce improvements in procedures, practices, and processes 
affecting multiple areas of SEA work. 


In addition to their capacity-building goals, Centers’ understanding of states’ specific needs for assistance in addressing the 
federal priorities was expected to shape their TA design and planning. The evaluation therefore examined how the Centers 
assessed SEA needs and developed work plans: 


Centers’ needs assessment and planning involved multiple method's but universally focused on interactive discussions with state 
education leaders and SEA staff. In design-focused interviews conducted in 2015, the Centers reported working closely with 
SEAs to jointly identify states’ needs and to co-develop work plans to help SEAs meet their goals.® All but one Center reported 
that meetings with SEA staff and chief state school officers were a primary means of needs assessment. Needs were also 
identified in some Centers through, for example, SEA requests (14 Centers), Centers’ own knowledge of existing needs (12 
Centers), review of educational data (9 Centers), needs assessment surveys (4 Centers), and embedding of Center staff in SEAs 
(4 Centers). All but two Centers used multiple forms of needs assessment, with processes continuing even after completing 
their work plans (see Appendix Exhibit D-3). 


Although 95% of Center projects served SEAs, they served other constituents as well, with more than one-third serving LEAs and 
26% serving Regional Centers. LEAs were the most frequently served project constituents other than SEAs (see Exhibit 3), and 
frequency of service to LEAs increased over time, according to Centers’ activity reports (see Appendix Exhibit D-4). While the 
U.S. Department of Education’s initial guidance for Centers directed them to provide TA directly to SEAs, guidance changed in 
2016 to encourage more direct TA to LEAs. In interviews, Center staff and TA recipients reported that, as part of building SEA 
capacity to implement policy at the local level, Center staff sometimes led or co-led LEA meetings or training sessions. With SEA 
staff present, Center staff modeled outreach and communication practices for SEAs while also providing direct services to LEAs. 
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As stated in the legislation, the Content Centers were required to build the capacity of Regional Centers as well as SEAs. Based 
on Centers’ activity reports, Content Centers sometimes served Regional Centers through TA calls or training sessions and 
provision of resources. 


Exhibit 3. Affiliations of TA Recipients of Center Projects 


Percentage of Projects of Each Center Type 

Recipient Type Regional Centers Content Centers 

N=188 N=145 
SEA 99% 90% 
LEA 43% 26% 
Regional Center 37% 
State level, non-SEA 
Content Center 
Other 29% 


Source: Center activity reports, 2015-2017. 

Percentages of projects serving these TA recipients in any project activities. 

Percentages can add up to more than 100% because projects could serve more than one constituent group. 
“Other’ recipients include institutions of higher education, research centers, and professional associations. 


2. HOW CENTERS IMPLEMENTED TECHNICAL ASSISTANCE 


To understand how the Centers carried out their capacity-building plans, the evaluation focused on TA strategies (and types of 
activities or service modes used to deliver the strategies) in two of the original seven federal priority areas — early learning and 
teacher/leader effectiveness. This approach allowed a more in-depth look at implementation while limiting burden on 
respondents. The evaluation also looked at the types and nature of Centers’ collaborations to understand how they were 
leveraging the broader network of federally funded TA in education to make efficient use of federal resources and to avoid 
duplication of efforts. Finally, the evaluation documented implementation challenges and how Centers addressed the 
challenges. (See Appendix D for further details on each of the implementation findings.) 


To implement projects in the two priority areas, Centers and TA recipients reported in interviews that Centers drew from a common 
set of TA strategies, which included thought partnering, cross-state sharing, coordination across divisions and agencies, 
stakeholder engagement, and modeling new practices. According to their activity reports, Centers then enacted these strategies 
through an array of TA activities that included consultations, meeting facilitation, provision of products or resources, and 
workshops. (See Appendix Exhibits D-5 and D-6 for interview data on Center strategies and activity report data on TA activities; 
see Appendix E for detailed examples of how these strategies are used within projects.) 


e = Thought partnering. Centers and TA recipients reported that Centers served as a “critical friend” regularly available to 
SEA staff for brainstorming and problem solving, especially to broaden or shift SEA perspectives. For example, a TA 
recipient on a project focusing on the state teacher evaluation system described how a Center helped the SEA define 
the problems of practice that the SEA could address in the near term to further their ultimate goal of creating a culture 
of improvement. 


e §=Cross-state knowleage building. Respondents reported that Centers shared common concerns and promising practices 
among SEAs. For example, one Center brought SEA staff of several states together for peer networking and problem 
solving. They also produced policy briefs based on scans of various state policies that addressed common problems or 
goals. 
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e  Cross-policy coordination. Respondents reported that Centers coordinated planning across different divisions or 
agencies, helping the SEA understand how policy areas intersect. For example, one Center helped bring together staff 
of the SEA’s early learning, educator effectiveness, and school improvement divisions to develop teacher evaluation 
methods. The Center helped SEA staff develop teacher performance measures appropriate to early learning. 


e = Facilitation of stakeholder engagement. Respondents described Centers facilitating various SEA communications, 
including meetings and workshops aimed at engaging stakeholders such as state and local education leaders, educator 
associations, and unions. For example, one Center assisted SEAs in developing outreach skills and materials, conducting 
web dialogues, and using surveys and listening sessions to obtain input and feedback on policy. 


e Modeling. Respondents reported that Centers often delivered a service with the expectation that the SEA would learn 
the process and ultimately replicate it. For example, one Center facilitated a task force meeting series and LEA 
outreach meetings for several months before turning them over to the SEA staff. 


Most Center projects implemented multiple TA activities to operationalize the key TA strategies above. The activities most 
frequently included in projects were consultations (68% of projects), meeting facilitation (50%), and provision of products and 
resources such as guidance documents, policy briefs, report templates, or policy scans (35%), workshops (29%), and brokering 
stakeholder connections (26%). Over the three years examined, 75% of projects (81% of Regional Center projects and 68% of 
Content Center projects) included at least two types of activities. (See Appendix Exhibits D-6 and D-7 for additional data on 
Centers’ TA activities and number of activities per project.) 


Most Center projects included at least one collaboration with another Center. Since the Comprehensive Centers were designed to 
work together and to expand the expertise available to SEAs, the evaluation examined the frequency and types of Center 
collaborations with each other and with other organizations. Survey respondents reported that most projects (79% of Regional 
Center projects and 72% of Content Center projects) included at least one collaboration with another Center. Collaborations 
occurred most often between Regional and Content Centers. As shown in Exhibit 4, among Regional Center projects, 70% 
included a collaboration with one or more Content Centers. Among Content Center projects, 67% included a collaboration with 
one or more Regional Centers. 


Exhibit 4. Collaboration Between Centers 


Regional 


Cross-collaboration 70% 
Center- 
Led Inter-collaboration 32% 
Projects 
N= 170 No collaboration 21% 
Content Cross-collaboration 67% 
Center- 
Led Inter-collaboration 35% 
Projects 
No collaboration 9 
N=107 28% 


Cross-collaboration: Project staff reported collaboration with at least one other center from a different center type 
Inter-collaboration: Project staff reported collaboration activities with one other center from the same center type 
Percentages totals can exceed 100% because projects may have both types of collaboration. 


Source: Center staff survey, 2015-2017. 


In cross-collaborations, a Regional Center might provide state-specific knowledge, while a Content Center provides content 
expertise and experience with other states. For example, in discussing a project on educator equity planning in two states, staff 
of a Regional Center described a collaboration with a Content Center. The collaboration involved co-facilitation of stakeholder 
engagement meetings to lead stakeholders through a structured process to identify causes of disparities in teacher quality. A 
Regional Center staff member reported: 
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I’ve been working with the [Content Center] on resources and communicating about how | can adapt 
them to meet the need of [the states] as the stakeholder engagement meetings have progressed. 
[Content Center and Regional Center staff] co-facilitated the analysis process ... [Content Center staff] 
had done it at that point with several states, and | hadn’t done it with any of the states I’ve worked 
with so far. 


In another example, a Content Center staff member commented that the Regional Center collaborator on an early learning 
project helped them to understand state needs and the state policy landscape, as well as the necessary connections to key 
stakeholders: “because [the Regional Center lead] is so grounded in the state and the department.” 


Inter-collaborations between Content Centers occurred when multiple areas of content expertise were required on a single 
project. For example, one project focused on early learning practices within turnaround schools. Regional Centers collaborated 
with each other to maximize their resources when addressing a common need experienced across several regions. For example, 
two Regional Centers collaborated on planning a webinar on teachers’ data use delivered to their two regions. 


The Centers also reported in surveys that they collaborated with other types of organizations, including Regional Educational 
Laboratories (22% of projects), institutions of higher education (20%), and other TA providers (29%). TA recipients reported in 
surveys that a particularly helpful service of the Centers was provision of access to experts from these various sources. 


Changing priorities at SEAs and turnover of SEA staff were the project challenges most frequently reported by Center staff. These 
were followed in order of frequency by the challenges of meeting diverse needs across SEAs, cuts in SEA staffing and budgets, 
project timeline constraints, and policy shifts in state government (see Exhibit 5). Center staff reported similar project 
challenges across the two types of Centers; however, Regional Center staff were more likely to report timeline constraints as a 
challenge than Content Center staff. This may be because, as reported in interviews, Regional Centers frequently described 
assisting SEAs in responding to legislation, and this work could be particularly rushed due to state political or fiscal pressures. 


Exhibit 5. Project Challenges Reported by Center Staff 


Changing priorities at an SEA 5256 
Turnover among state education agency 
(SEA) staff 48°55 
Meeting diverse needs across SEAs 40-47 
(and/or regions) 
Cuts in SEA staffing/budgets 4144 
Project timeline constraints 24 49 
Policy shifts in a state government 3640 
Turnover in chief state school officer 31 37 
and/or other SEA leadership 
Lack of communication withinan SEA 3334 
0% Percentage of projects 100% 
Regional Centers Content Centers 


Source: Center staff surveys, 2015-2017. 
Project challenges reported by at least one staff member for each project. N = 171 Regional Center projects, 108 Content Center 
projects. 


Most TA recipients did not report any challenges in working with Centers, but those that did most often identified scheduling 
difficulties (14%), staff turnover (9%), and an unclear understanding of the Center’s role (8%). TA recipients of the two Center 
types reported similar challenges (see Exhibit 6). In interviews, TA recipients agreed with the Centers that turnover and priority 
shifts could cause disruption and delays. For example, one said: 
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| worry because [the SEA] is so lean ... nothing's more important than some of the big levers that we've 
developed to support educator effectiveness. But once it's passed, and once something else comes 
along as a priority, we lose the capacity to do these things well to sustain them, and ! worry about that 
lost opportunity. 


Two other TA recipients each elaborated in interviews on the challenge of learning how to use the Centers: 


| have a lot of TA providers ... that want us to come up with work for them to do, and | just don’t have 
time ... [I’m] just trying to figure out what their roles are. 


It takes time to realize what their strengths are and to use them in ways that benefit. 


Exhibit 6. Challenges in Working with Centers Reported by TA Recipients 


Percentages Reporting Challenge 


TA Recipients of 
Content Centers 


TA Recipients of 
Total Regional Centers 


Challenges Reported by TA Recipients in 2017 =231 N=115 N=116 
Difficulty scheduling time for our staff to participate 14% 
Staff turnover at our organization 6% 
Unclear understanding of role of the Center 6% 
Misalignment between our needs and Center priorities 7% 


Source: TA recipient survey, 2017. 
See Appendix Exhibit D-11 for full 2017 results as well as 2015 and 2016 results. 


To address the challenges of turnover and priority shifts, the Centers emphasized in interviews being persistent, building broad buy- 
in, and maintaining flexibility. 


e = Being persistent. The Centers followed up in a timely manner to maintain constituents’ focus while accommodating their 
schedules. Given tight budgets and understaffing, exacerbated by turnover and competing priorities, SEA staff reported 
that they sometimes had difficulty sustaining focus on their ongoing work with Centers. Centers reported addressing this 
challenge by maintaining communication with the SEAs throughout slow periods, including regular check-ins, while flexibly 
accommodating SEAs’ need to shift project timetables and priorities. 


e = Building broad buy-in. The Centers built broad stakeholder buy-in to project goals, within and beyond the SEA to include 
LEAs and other state and local agencies. Center respondents said that their success depended on commitment at multiple 
levels of SEA staff, as well as from LEA leaders and other stakeholders across the states. They described how, in helping 
SEAs build broad stakeholder support for projects, Centers fostered long-term investment in project goals and ongoing 
implementation. They further noted that relationships with mid-level staff were particularly valuable given frequent 
turnover at the leadership level. Relationships beyond the SEA, with additional state and local agencies, added stability as 
well. As one Center staffer said: 


When we think about building capacity, we’re always kind of expanding the conversation away 
from just the agency itself to the broader network, because we know there will be turnover. 


e = Maintaining flexibility. Many Centers responded to challenges with flexibility, modifying projects in response to shifts in 
policy focus, funding, and the needs of constituents. In interviews, Centers reported modifications in project plans in 
response to new SEA priorities, new federal legislation such as the Every Student Succeeds Act, or emerging stakeholder 
input. For example, a TA recipient of one project indicated the value of the Center’s flexibility: 
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We got a new Commissioner ... the goals of the agency have probably changed because of that 
change in leadership. And the Comprehensive Center has been able to work with us to adjust and 
do things that we needed to do to address those goals. 


3. WHAT OUTCOMES CENTERS ACHIEVED 


Overall, Centers and TA recipients positively rated Centers’ contributions to capacity, with the highest ratings given for knowledge 
and skills. Center staff rated the contribution of most projects (72%) to constituents’ capacity between “to a moderate extent” 
and “to a great extent,” as illustrated in Exhibit 7. (See Appendix D for more details on each of the findings related to Center 
project outcomes.) 


Exhibit 7. Center Staff Ratings of the Extent to which Center Projects Contributed 
to the Overall Capacity of Constituents 


= eee _— mw 
acacia —_—- ee a 


De ae — Kw 


fia Between toa i Between to some Between none 
moderate and to extent and toa and to 
a great extent moderate extent some extent 
(2.1-3) (1.1-2) (0-1) 


Source: Center staff survey, 2015-2017 


TA recipients also rated Centers’ overall contributions to capacity. The distribution of ratings on the 2017 survey is presented in 
Exhibit 8 for each of the four types of capacity that Centers identified. (See Appendix Exhibit D-12 for 2015 and 2016 results.) TA 
recipients gave generally positive ratings for Centers’ contributions to capacity. For example, 98 percent reported that on 
average, Centers built their knowledge and skills capacity to at least some extent, and 91 percent reported that result for 
organizational capacity. A minority of TA recipients had less favorable views, with 15 percent reporting that on average, Centers 
did not build their capacity for policy design at all or at most only to some extent, and 17 percent reported that result for policy 
implementation capacity. Overall, respondents rated contributions to knowledge and skills, on average, a score of 2.5 ona scale 
from 0-3: at the mid-point between “to a moderate extent” and “to a great extent.” They rated contributions to organizational, 
policy design and implementation capacity, on average, a score of 2.1, approximately “to a moderate extent.” 
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Exhibit 8. TA Recipient Ratings of the Extent to Which Centers Contributed to Specific Types of Capacity 


Percentage of TA recipients giving average ratings: 


Between toa Between to some Between none 
moderate extent and to a and to some 
extent and toa moderate extent extent 
great extent (1.1-2) (0-1) 
Source: TA recipient survey, 2017. (2.1-3) 
Ns range from 201 to 227. 


In interviews, TA recipients attributed the following specific capacity changes to the Centers: 


e Knowledge and Skills. A TA recipient of an early learning project said the Center presented research that enhanced 
“knowledge of staff [of SEAs and other agencies on a task force] on strategies that can be employed to alleviate the need 
for an expulsion or suspension” in prekindergarten classrooms. 


e Organizational Capacity. A TA recipient of a teacher/leader project said that the Center brought together divisions including 
school improvement, special education, and English language learners: “[The Center] has been instrumental ... We have 
been able to slide the scale a little bit away from being siloed and more towards being a collaborative agency. We rewrote 
our ... interactive action plan process for ESSA this year as a collaborative group.” 


e Capacity for Policy Design. ATA recipient reported that the SEA increased its capacity to help form policies on educator 
certification, working with the Board of Education: “We brought in studies that [the Center staff] had put together ... We 
had assistance from Center staff in developing the presentations and the activities. The result was a very fruitful board 
meeting where the Board adopted significant changes to the rules related to teacher preparation.” 


e Capacity for Policy Implementation. On one early learning project aiming to encourage statewide administration of a 
kindergarten readiness assessment, a TA recipient commented that the Center provided SEA staff with a “data-driven 
dialogue” method that they now use: “We have taken it out to the [kindergarten entry inventory] implementing folks and 
have done the trainings on the ground. We’ve actually used it here at the state level as well in some of our own data-driven 
capacity.” 


More than half of TA recipients in 2017 reported using the information provided by the Centers to make recommendations (64%), 
develop new processes or programs (63%), and draft internal memos or reports (57%). Slightly less than half of TA recipients 
(46%) reported using the information to overcome a barrier or challenge (see Exhibit 9). Most TA recipients (between 86% and 
88% each year) reported taking at least one of these actions using materials provided by the Centers (see Appendix Exhibit D- 
18). 
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Exhibit 9. TA Recipient Actions Reported 


Percentage of TA Recipients 


Regional Content 
TA Recipient Actions Using Materials Provided by the Centers, 2017 ig 
Total Centers Centers 


N=231 N=115 N=116 
Made recommendations to SEA leadership or staff 64% 61% 66% 
Used the information in developing new processes/projects/programs 63% 69% 58% 


Used the information in drafting internal memos/plans/reports 57% 64% 50% 


Used the information to overcome a barrier or challenge 46% 50% 42% 


Reported at least one of the above actions 


Source: TA recipient survey, 2017. 
See Appendix D for 2015 and 2016 TA recipient survey results. 


Center staff and TA recipients reported in interviews that Center projects either achieved goals, exceeded goals or, for projects that 
had not yet concluded, were on track to achieving goals. Of 27 projects for which questions about goal achievement were 
addressed by both Center staff and TA recipients, both types of respondents agreed that goals had been achieved for 18 
projects, and both agreed that the project was on track towards achieving its goals for 7 projects. In discussing project 
achievements, Center staff focused on outcomes related to building constituents’ capacity. TA recipients often focused on the 
Centers’ role in helping them achieve broad state policy or reform goals, for example: 


Based on the expert feedback [from the Center], | made adjustments to a proposal for future direction 
of the teacher evaluation model. The feedback ensured a much higher quality product to submit to 
leadership. 


Updating the Child Development Permit structure has been very challenging. | don't believe we could 
have achieved its results without the assistance of the [Center]. 


Most TA recipients (85% or more each year) reported in surveys that the Centers enabled them to accomplish something they 
might not otherwise have been able to do(see Exhibit 10). 


Exhibit 10. TA Recipient Accomplishments Enabled by the Centers 


Percent of TA recipients reporting the Center’s TA enabled them to 
Survey Year accomplish what they may have not otherwise accomplished 


2015 (N = 207) 
2016 (N = 210) 
2017 (N = 230) 

Source: TA recipient survey, 2015-2017. 


Center evaluation reports indicate that all Centers included at least a few questions about capacity building in the surveys and/or 
interviews administered by Center evaluators for annual reporting. Centers or their evaluators usually administered surveys and 
conducted interviews annually or as a follow-up to events. To meet federal reporting requirements, Centers focused these 
instruments on questions about their TA quality, relevance, and usefulness, but they often included questions focused on 
capacity-building outcomes as well. In interviews, Centers also reported using informal methods, such as ad hoc observations 
and conversations with constituents, to identify and describe project capacity-building outcomes. 
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TA strategies and activities reported to be effective were those that reflected capacity-building principles: imparting skills, 
practices, or tools that fostered ownership, process change, and/or long-term, sustainable change. |n interviews, TA recipients 
identified Centers’ products and tools, thought partnering, facilitation, and cross-state knowledge building as particularly 
effective in producing project capacity-building outcomes. TA recipient survey results also identified Centers’ products and 
tools, as well as access to experts, among particularly helpful services. 


As an illustration of how a TA strategy worked effectively to build capacity, both Center staff and TA recipients described how 
Center products and tools contributed to a sense of ownership among SEAs, process change, and long-term change. Products 
and tools identified as effective included those that served as models or templates adaptable to different scenarios and could 
be used by SEA staff to improve agency processes, including internal performance and external communication with LEAs and 
other stakeholders. For example, a Center staff member of one early learning project explained that the Center developed 
model papers on school readiness indicators, detailing how the indicators could guide instructional strategies. SEA staff used 
these as templates for writing papers on additional indicators and disseminating them to LEAs. ATA recipient of a 
teacher/leader project reported that the Regional Center working with her had provided performance rubrics that helped to 
build organizational capacity. The rubrics were used for agency restructuring “and will affect every single program in the 
agency.” 


In surveys, Center staff identified their content expertise and knowledge of state context (75-87% of projects), as well as their 
strong relationships with SEAs and state educational leaders (58-62%), as among the most significant supports for their 
projects’ success. Other important supports reported by Center staff were SEA and Center leadership commitment (48-49%) 
and Center collaborations (33-39%). 


4. IMPLICATIONS FOR THE COMPREHENSIVE CENTERS PROGRAM 


Overall, the evaluation found that Centers shared similarities in their approaches to the design and implementation of their 
work, and Centers and key TA recipients reported that the work generally helped build SEA capacity. Profiles of two Center 
projects are provided in Appendix E as a supplement to this report. These two projects are not necessarily representative of all 
Center projects, but were selected to bring the overall findings in this report more to life while also recognizing the unique 
combinations of needs, strategies, challenges, and outcomes that may make up each project. 


This evaluation’s findings are consistent with the findings of the prior national evaluation of the Centers, published in 2011. The 
factors that respondents highlighted as instrumental to achieving desired outcomes are worth documenting for guidance to 
future Centers. The evaluation also uncovered some common challenges to the Centers’ work and points to possible ways to 
address these challenges moving forward. In this concluding section, we fill in the “black boxes” in Exhibit 1 with our main 
findings on how Centers design and implement their work and how they achieve their outcomes (see Exhibit 11). 
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Exhibit 11. How Comprehensive Centers Build SEA Capacity 


Dimensions of Capacity Principles of Capacity Building 
Knowledge and skills Fostering: 
Organizational capacity * Ownership 
Capacity for policy development or design * Long-term change 
Capacity for policy implementation * Organizational processchanges 


Needs Assessment and Planning 

* Discussionswith SEA staff and chief state school officers 

* Jointly identify needs and co-develop work plans 

* Multiple forms of ongoing needs assessment 

* Varied constituents including SEAs, LEAs, Regional Centers, and others 


TA Strategies Collaborations 

* Thought partnering * Between Regional and Content 

* Cross-state knowledge building Centers 

* Cross-policy coordination * Between Centers of the same type 
* Stakeholder engagement 


Modeling Challenges: ChangingSEA priorities, 


turnover of SEA staff, meeting diverse 


TA Activiti needs, time constraints 


Consultations 

Meeting facilitation 

Provision of products or resources 
Workshops 


Responses to Challenges 

* Persistent, timely follow-up 

* Broad stakeholder buy-in 

* Flexibility and project modifications 


S8WOIINO ~¢——— uoljejUawWajdw); <———— u@iseq 


From these findings emerge a number of capacity-building levers that are potentially important in enabling Centers to overcome 
challenges and produce longer-term capacity changes. Future Center operators and Department of Education program staff may 
wish to adopt and encourage these practices that offer potentially important levers for capacity building. These promising 
capacity-building levers are presented in Exhibit 11 and include: 


e = Engaging a broad array of stakeholders. Throughout the evaluation, respondents emphasized that the Centers played a 
particularly important role in reaching out to, bringing together, and gathering input from multiple stakeholders both within 
and beyond SEAs, and in guiding SEA staff to learn and take over this role. Centers also found that engaging diverse 
stakeholders, at multiple levels within SEAs and other agencies, helped build continuity in new practices throughout SEA 
staff and leadership turnover. 


e = Providing products and tools that SEAs could use on their own. TA recipients emphasized the usefulness of Centers’ products 
and tools that SEAs could use or adapt for internal and external communication and guidance, increasing their capacity to 
lead and implement organizational and policy changes. 


e = Identifying and imparting resilient practices. Centers and TA providers noted that despite the continuing challenges of staff 
and leadership turnover and shifting priorities, Centers could build long-term capacity by helping SEAs identify and use 
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practices that could endure through turnover and policy shifts. These practices include improving coordination and 
communication across divisions and agencies, connecting with other states, and engaging stakeholders. 


e = Being flexible in implementation of plans. The greatest challenges to the Centers’ work came from turnover and changes in 
state leadership, staffing, and policy. Center staff and TA recipients noted that Centers were able to continue to work 
towards long-term SEA capacity change by adjusting their plans, schedules, and strategies in response to continually 
changing priorities and needs. 


While respondents reported generally positive experiences with the Centers, a few areas for program improvement emerged that 
program staff may wish to consider: 


Further clarify the Centers’ role and expected outcomes at the local level, Atthough Centers’ charge was focused on SEAs, the 
ultimate goal of the SEAs is to improve outcomes at the local level. Centers worked directly with LEA and school staff on 
some projects, often as part of modeling and guiding the SEAs to work with LEAs themselves. Over the course of the grant 
period, Center projects were increasingly likely to involve LEAs directly. The U.S. Department of Education guidance to 
Centers shifted in 2016 towards encouragement of work with LEAs. It may be helpful to further develop the concept of 
Center capacity building to include more specific goals and expected outcomes at this level. 


e = Provide additional guidance for SEAs on how to make the best use of Centers. Although some SEA staff had a history of 
working with the Centers and a thorough understanding of the Centers’ role, a few reported that they would benefit from 
receiving more detailed information about what services Centers do and do not provide, and how the scope of their TA is 
distinct from that of other federal TA providers. 


Finally, SEAs expressed concern during interviews about the sustainability of their initiatives without continuing support from 
the Centers. Center staff and TA recipients often noted that time pressures and limited SEA staffing levels impeded progress 
toward their policy goals, leading them to rely heavily on Center support. TA recipients commented that when responding to 
urgent requests, the Centers were able to quickly provide resources (such as research supporting a policy proposal) that SEA 
staff did not have time or sufficient expertise to access on their own. Even with their capacity bolstered as intended, SEAs may 
be limited by insufficient resources and staff, and may thus require continued Center support. 


Looking Ahead 


It is important to emphasize that this evaluation is not able to measure the causal impacts of the Centers’ work and that 
much of the findings are based on the perceptions of Center staff and TA recipients identified by the Centers. The findings 
should therefore not be used to make summative judgments about the Comprehensive Centers program but rather as 
insights into how the program is playing out in practice and possible avenues to help it better meet program objectives in the 
future. 


Evaluating the Centers’ TA was challenging because it is difficult to isolate the effects of their TA from other influences on the 
actions and capacity of SEAs and to clearly link the Centers’ work to student, teacher, and school outcomes. TA is not a 
specific intervention but a set of strategies and services that are customized and responsive to each constituent’s needs. 
Further specification of high-impact capacity-building outcomes is needed, as are measures of SEA capacity for working 
effectively with LEAs. Validated measures based on the capacity-building definitions and indicators examined in this 
evaluation would improve future evaluations and support a range of stakeholders in program improvement efforts. 
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ENDNOTES 


1 http://www?2.ed.gov/programs/newccp/legislation.html 


277 FR 33563. https://www.federalregister.gov/articles/2012/06/06/2012-13735/applications-for-new-awards-comprehensive- 
centers-program#h-4 


3 The seven priority areas in more detail are (1) implementing college and career-ready standards and aligned, high-quality 
assessments for all students; (2) identifying, recruiting, developing, and retaining highly effective teachers and leaders; (3) turning 
around the lowest-performing schools; (4) ensuring the school readiness and success of preschool-age children and their 
successful transition to kindergarten; (5) building rigorous instructional pathways that support the successful transition of all 
students from secondary education to college without the need for remediation, and careers; (6) identifying and scaling up 
innovative approaches to teaching and learning that significantly improve student outcomes; and (7) using data-based decision- 
making to improve instructional practices, policies, and student outcomes. Some Centers also addressed Native American 
Education and special needs. 


477 FR 33563. https://www.federalregister.gov/articles/2012/06/06/2012-13735/applications-for-new-awards-comprehensive- 
centers-program#h-4 


° Program performance requirements directed Centers to produce high quality, relevant, and useful services, based on 
constituent ratings. Centers’ performance reporting and internal evaluations drew on surveys of constituent satisfaction with 
these aspects of Centers’ TA. 


® See appendix pages A-4 to A-7 for context and background on SEA needs for TA. 
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